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Abstract. A partition of {!,...,«} has an m-nesting if there exists ii < i2 < ■ ■ ■ < im < 
jm < jm-i < • ■ ■ < Ji: where ii and ji are in the same block for all 1 < / < m. We 
use generating trees to construct the class of partitions with no m-nesting and determine 
functional equations satisfied by the associated generating functions. 

We use algebraic kernel method together with a linear operator to describe a coefficient 
extraction process. This gives rise to enumerative data, and illustrates the increasing com- 
plexity of the coefficient formulas as m increases. 



1. Introduction 

In this work we address the enumeration of set partitions that avoid a particular class 
of patterns. The patterns considered here, known as m-nestings, arise in a graphical repre- 
sentation of set partitions. Our goal is to determine useful enumerative information about 
partitions that contain no m-nesting. This work is in the context of recent studies of other 
combinatorial objects that avoid similar or related patterns, in particular in the study of pro- 
tein folding [7]. Our strategy parallels a recent generating tree approach used by Bousquet- 
Melou to enumerate a class of pattern avoiding permutations [3j. Here, in a less condensed 
form, upon exhibiting a general functional equation for all m, we apply both an appropriate 
transformation of variables and a multiplicative factor to produce symmetry in the kernel 
of the functional equation. This permits us to apply the algebraic kernel method and gen- 
erate a telescoping sum, which greatly simplifies the expression. The enumerative formulas 
are obtained by coefficient extraction. The result of the analysis produces a first look into 
how such numbers are composed in a recurrence. It is our hope that readers new to the 
generating tree approach for obtaining multivariate functional equations could understand 
the enumerative power and closed form limitations of this method. 

1.1. Notation and definitions. A set partition tt of [n] := {1, 2, 3, . . . , r?,}, denoted by 
TT e n„, is a collection of nonempty and mutually disjoint subsets of [n], called blocks, whose 
union is [n]. The number of set partitions of [n] into k blocks is denoted S{n,k), and is 
known as Stirling number of the second kind. The total number of partitions of [n] is the 
Bell number Bn = X]fc 'S'(^) k). We represent tt by a graph on the vertex set [n] whose edge 
set consists of arcs connecting elements of each block in numerical order. Such an edge set 
is called the standard representation of the partition tt, as seen in [6J. For example, the 
standard representation of 

1|2568|37|4 

is given by the following graph with edge set {(2, 5), (5, 6), (6, 8), (3, 7)}: 
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With this representation, we can define two classes of patterns: crossings and nestings. 
An m-crossing of tt is a collection of m edges ji), {12, j2), ■ ■ ■ , iim)jm) such that ii < 
12 < ■ ■ ■ < im < ii < 32 < ■ ■ ■ < im- Using the standard representation, an m-crossing is 
drawn as follows: 




h h ■ ■ ■ im jl 32 " • jm 



Similarly, we define an m-nesting of vr to be a collection of m edges (ii, ji), (^2,^2), • • • , 
{im,jm) such that ii < 12 < ■ ■ ■ < im < 3m < jm~i < ' ' ' < jl- This is drawn: 




ii «2 • • • ik jk • • • j2 jl 

A partition is m-noncrossing if it contains no m-crossing, and it is said to be m-nonnesting 
if it contains no m-nesting. 

1.2. Context and plan. Chen, Deng, Du, Stanley and Yan in p] and Krattenthaler in [8] 
gave a non-trivial bijective proof that m-noncrossing partitions of [n] are equinumerous with 
m-nonnesting partitions of [n], for all values of m and n. A straightforward bijection with 
Dyck paths illustrates that 2-noncrossing partitions (also called noncrossing partitions) are 
counted by Catalan numbers. Bousquet-Melou and Xin in |1] showed that the sequence 
counting 3-noncrossing partitions is P-recursive, that is, satisfies a linear recurrence relation 
with polynomial coefficients. Indeed, they determined an explicit recursion, complete with 
solution and asymptotic analysis. They further conjectured that m-noncrossing partitions 
are not P-recursive for all m > 4. Bell numbers are well known not to be P-recursive 
because of the composed exponentials in the generating function B{x) = e^"^'^ as explained 
in Example 19 of [2|. 

Since m-noncrossing partitions of \n\ and m-nonnesting partitions of [n] are equinumerous, 
we study m-nonnesting partitions in this paper and show how to generate the class using 
generating trees, and how to determine a recursion satisfied by the counting sequence for 
m-nonnesting partitions. 

Our approach is heavily inspired by Bousquet-Melou's recent work on the enumeration 
of permutations with no long monotone subsequence in [3]. She combined the ideas of 
recursive construction for permutations via generating trees and the algebraic kernel method 
to determine and solve functional equations with multiple catalytic variables. 

In Section [21 we employ Bousquet-Melou's generating tree construction to find functional 
equations satisfied by the generating functions for set partitions with no m-nesting. The 
resulting equations, though similar to the equations arising in need a similar multiplica- 
tive factor but a different transformation of variables before a comparable analysis using 
algebraic kernel method techniques is applied. To succeed in obtaining information after 
applying the algebraic kernel method, a coefficient extraction procedure is required. This 
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is completed in Section |3l Unfortunately for us, unlike her work, we can only find explicit 
equations parameterised by m, with m catalytic variables for all m > 1 without obtaining 
a closed form expression for the number of set partitions of size n avoiding an m-nesting 
for m > 3. In the case of m = 3, however, it is similar to the functional equation given by 
Bousquet-Melou and Xin. 

We are able to provide new enumerative data for m > 4, and also offer evidence supporting 
the non-P-recursive conjecture of Bousquet-Melou and Xin in Section O 

2. Generating Trees and Functional Equations 

The generating tree construction for the class of m-nonnesting partitions is based on a 
standard generating tree description of partitions, and the constraint is incorporated using 
a vector labelling system. The generating tree construction has an immediate translation to 
a functional equation with m-variate series. 

2.1. A generating tree for set partitions. Let tt be a set partition. Define ne(7r) to be 
the maximal i such that tt has an z-nesting, also called the maximal nesting number of tt, and 
let ni™'' be the set of partitions of [n] for n > (where n = means the empty partition) 
with ne(7r) < m, thus (m + l)-nonnesting. 

Note that an arc over a fixed point is not a 2-nesting, but a 1-nesting. 

i j k 

We next describe how to generate all set partitions via generating trees in the fashion 
of [2]. First, order the blocks of a given partition, tt, by the maximal element of each block 
in descending order. 

Example 1. The first block of 1|2 5 6 8|3 7|4 is 2 5 6 8; the second block is 3 7; the third block 
is singleton 4; and 1 is the last block. Using the standard representation. 




bl. 4 bl. 3 bl. 2 bl. 1 



we number the blocks in descending order (from the right to the left) according to the 
maximal element in each block (that is, the rightmost vertex of each block). 

With the order of blocks thus defined, we warm up by generating all set partitions without 
nesting restriction first. Figured] contains the generating tree for all set partitions, in addition 
to the generating tree for the number of children of each node from the tree of set partitions 
to indicate how enumeration can be facilitated. 

(1) Begin with as the top node of the tree. It has only one child, so the corresponding 
node in the tree for the number of children is labelled 1. 

(2) To produce the n + 1st level of nodes, take each set partition at the nth level, and 
either add n + 1 as a singleton, or join + 1 to block j for each 1 < i < if the set 
partition has k blocks. 

3 



Summarizing the description above in the notation of |2] , we recall that the rewriting rule 
of a generating tree is denoted by: 

[(so), {{k) (ei,fc)(e2,fc) • • • {ek,k)}], 

where Sq denotes the degree of the root, and for any node labelled k, that is, with k descen- 
dents, the label of each descendent is given by (ej^k) for I < j < k. Thus, the class of set 
partitions has a generating tree of labels given by [(1) : (k) — ?■ (/c + 1)(A;)'^~"'^]. 



1 2 



1 2 3 1 2 3 1 2 3 



1 2 



12 3 12 3 



4 3 3 3 2 



Figure 1. Generating tree for set partitions and its corresponding generating 
tree of the number of children 



2.2. A vector label to track nestings. Note that in Figured! the generating tree of set 
partitions generates all set partitions graded by n, the size of vr but it does not keep track 
of nesting numbers. Also note that in the generating tree for the number of children, the 
number of children of tt is one more than the number of blocks of tt for any partition tt. 
Fix m. In order to keep track of nesting numbers, we need to define the label of vr G n'-™'-'. 

Definition 1. Define the label of a partition, L{n) = (ai(7r), a2(7r), . . . , am(7r)), or in short, 
L(7r) = (ai, 02, . . . , am) as follows. For I < j < m, 

{1+ number of blocks in tt, if vr is j-nonnesting, 

1+ number of blocks ending to the right of Q^j^g^^jgg 
the smallest vertex in the rightmost j-nesting 

By the rightmost j-nesting, we mean the minimal element in the j-nesting of a particular 
partition tt that is greater than or equal to all minimal elements in all j-nestings of tt. 

Example 2. To continue the example, let vr = 1|2 5 6 8|3 7|4 and suppose m = 3. Then 
L(l|2 5 6 8|3 7|4) = (3,4,5) for the following reasons. The rightmost 1-nesting is the edge 
with largest vertex endpoint, (6,8). Hence, ai(7r) = 3 because blocks 1 and 2 end to the 
right of vertex 6. The rightmost 2-nesting is the set of edges {(5, 6), (3, 7)} hence 02 (tt) = 4 
because 3 blocks end to the right of vertex 3. Finally, a3(7r) = 5 because the diagram has no 
3-nesting, and is comprised of 4 blocks. Note that in this convention, the empty set partition 
has label (1,1,..., 1), since it has no nestings and no blocks. 
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A set partition in n*^™) will have children. This is one plus the number of blocks, if 
there is no m- nestings (and hence no risk that adding an edge will create an m + 1-nesting). 
Otherwise, it indicates one plus the number of blocks to which you can add an edge without 
creating an m + 1-nesting. The label of a set partition is sufficient to derive the label of each 
of its children, and this is described in the next proposition. Also, remark that the label is 
a non-decreasing sequence, since the rightmost j-nesting either contains the rightmost j — 1 
nestings or is to the left of it. 

Proposition 1 (Labels of children). Let ir be in the set of set partitions on [n] avoiding 
m + 1-nestings, and suppose the label of n is L^n) = (01,02, . . . ,0^). Then, the labels of the 



Om set partitions of n^"^\ obtained by recursive construction via the generating tree 



are 



and 



(oi + 1, 02 + 1, . . . , Om + 1) (Add n + 1 as a singleton to n) 

( 2, 02, 03,..., (im-i^dm) (Add n + 1 to block 1) 

( 3, 02, 03,..., am-i,am) (Add n + 1 to block 2) 

( ai, 02, 03,..., (im-i,(im) (Add n + 1 to block ai — 1) 

(oi + 1, oi + 1, 03, ... , Om-ii dm) (Add n + 1 to block ai) 

(ai + l,ai + 2, 03,..., (im-i,(im) (Add n + 1 to block ai + 1) 

(oi + 1, a2 + 1, 02 + 1, . . . , (im-i, dm) (Add n + 1 to block 02) 

(oi + 1, 02 + 1, as + 1, . . . , ttm-i + 1, Om-i + 1) (Add n + 1 to block Om-i) 

(ai + 1, 02 + 1, as + 1, . . . , ttm-i + 1, cim) (Add n + 1 to block am — 

Proof. By careful inspection. □ 

(3) 

Example 3. Consider the following partition from Hg . The reader can refer to its arc 
diagram in Example 1 which shows that is is 3-nonnesting, thus also 4-nonnesting. The 
partition 1|2 5 6 8|3 7|4 with label (3, 4, 5) has five children and their respective labels are: 

TT L{7r) 

1|2568|37|4|9 (4,5,6) 

1|25689|37|4 (2,4,5) 

1|2568|379|4 (3,4,5) 

1|2568|37|49 (4,4,5) 

19|2568|37|4 (4,5,5) 

Notice that in Proposition [H the first label comes from adding n + 1 as a singleton to 
get an element of n^'^\. The other a^ — 1 labels result from adjoining the element n + 1 
to the maximal element of block I for every 1 < / < am — 1 blocks without creating an 
m + 1-nesting. 

Example 4. As we mentioned before, 2-nonnesting set partitions are counted by Catalan 
numbers. The generating tree construction given in Proposition [T] restricted to this case is 
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given by 

[(l):(A:)->(2)(3)...(fc + l)], 

which is the same construction for Catalan numbers given in [2]. The generating tree for 
3-noncrossing partitions is given by 

[(1, 1) : (z, j) + + 1)(2, j)(3, j) • ■ ■ {i,3){t + 1,1 + m + l,i + 2)...{z + 1, j)]- 

2.3. A functional equation for the generating function. The simple structure of the 
labels of a partition's children in Proposition [T] permits a straightforward translation of the 
combinatorial construction into a functional equation. Let us define F{ui,U2, ■ ■ ■ ,Um',t) to 
be the ordinary generating function of partitions in n^™-* counted by the statistics ai, 02, 
. . . , ttm and by size, 



am. 

Uo ... LLjn ) 



ai,a2,...,a„ 



where Fg,{t) is the size generating function for the set partitions of n*^"^) with the label 
a = (oi, 02, . . . , Urn)- Thus, when m = 2, 

F(u; t) = UiU2+Ui\2^t+{ui^U2^ + Ui\2^) t^+{ui\2'^ + 2 ^1^2^ + Ui^U2'^ + Ui^U2^) t^ + . . . ■ 

Proposition [T] implies 

F{ui,U2, ...,Um;t) = U1U2 . . .Um + tUiU2 ■ ■ ■ UmF{ui,U2, . . . ,Um]t) 



a\,a2,...,am a=2 

m ij 



ai+l ao+l aj_i+l a ij+i a 



L2 ■ ■ ■ U.J • • • • 

ai,a2,...,am 3=2 a=aj_i+l 

We compactify the sums using finite geometric series sum formula and summarize the 
above derivation into the following functional equation for the generating function. 

Proposition 2. 

(1) F(u; t) = U1U2 . . .Um + tUiU2 . . . UmF{u; t) 

j^^^ f -^(u; t) - UiF{l, U2,..., Um] t ) 
^ \ Ml - 1 

+ tYu.U2...U, y --^ 

where -F(u; t) = F{ui,U2, . . . , Um', t). 



3. Processing the functional equation 

To process the functional equation for F(u; t) we transform the variables to get a form 
more amenable to analysis. We follow |3] , and do this in two steps. The first rewrites in a. v 
variable to remove the exponent restriction on the -Uj's because ai(7r) < 02 (tt) < ■ ■ ■ < am(7r); 
the second transformation to a set of x variables in the next section allows us to analyse the 
coefficients. 

3.1. Removing the exponent restriction. Define 

7rgn{'") 

where (ci, 02, . . . , a^) = L^tt). Thus, we have eliminated the dependency ai < a2 < ■ ■ ■ < 
between the exponents of mi, ^2, • • • , "^m in F{u; t). 
We can write F in terms of F and vice versa: 

F[Vi, Vm+U t) = F [—,—,... , ; Vm+lt , 

and 

F{ui, ...,Um; Vm+lt) = F{ui^rnVm+l,U2,mVm+l, • • • , UmVm+1 , Vm+l, t) , 

where Uj^m = v-jUj^i . . . u^- The function F satisfies a simpler functional equation. 

Proposition 3. The generating function F{y\ t) = F{vi,V2, . . . , Vm, Vm+i] t) of set partitions 
o/n^™^ satisfies 



Vi , ^ F(\:t) 

- Vm+ltVi 



(2) F(v; t) = + tvi f F(v; t) + v^+i V 

■Vi F{V2,V2,V3, . . .,Vm+l]t) 



f 2 Vi - f 2 

F{Vi, . . . ,Vj_i,Vj+i,Vj+i,Vj+2, ■ ■ ■,Vm,Vm+i; t) 



j=2 ^3 ~ ^J'+l 



The series F{1, 1, . . . , 1; t) is the generating function for the class of (m + l)-nonnesting set 
partitions. 

3.2. A second transformation. We take the functional equation for F(v; t) in Proposition 
|3] and rearrange the terms to find the kernel of the functional equation as follows 



(3) 



il-tvi- tviVm+i y~] I F{v; t) 



Vl , /Vi F{V2,V2,V3,. . . ,Vm+i;t) 
Vm+ltVi f 



Vm+1 ^V2 V1-V2 

F{Vi, . . . ,Vj_i,Vj+i,Vj+i,Vj+2, ■ ■ ■,Vm,Vm+i;t) 



i=2 



Vj - Vj+i 



The kernel is 



m ^ 
1-tVi- tViVm+1 . 



To exploit invariance properties of the kernel, we introduce the following transformation of 
the Vj's: 

Vm+l = 1 

V2 = I + Xm -\ \- X2 

Vl = 1 + Xm -\ \- X2 + Xi. 

This transformation enables us to rewrite the kernel as 



1 - t (xi + X2 + ■ ■ ■ + x„ + 1) ^1 + ^ ^ j . 



This new kernel is invariant under &m, the symmetric group on [m]. To simplify presentation 
of the functional equation, we use 

S = Xi + X2-\ hXm + l, ri = \ h 1 hi. 

X\ X2 '^m 

Divide both sides of the functional equation by s we just defined, we get 

(4) (- -th)F{xi,X2, ■ ■ ■ ,Xm]t) = 

s 

^ _ J S F{0,X2,Xs, . . . ,Xm;t) 
\S—Xi Xi 



j=2 



+E 

where 

F{xi,X2,...,Xm;t) = F{vi,V2,. . . ,Vm,Vm+i;t), 

and 

F(0,0,...,0;t) = F(l,l,...,l;t) 

is the evaluation that yields the ordinary generating function in t for set partitions avoiding 
m + 1- nestings. 

3.3. A multiplicative factor and a telescoping sum. We introduce a multiplicative 
factor, M(x) := xix^x^ . . . x^, to be applied to Equation (j4]). Let the new kernel, i^(x; t) be 
defined by \ — ht. Since the kernel -R'(x; t) is invariant under when we take the signed or- 
bit sum of the functional Equation (jl]) under &mi namely, Ylia^&m ^ (cr)^(functional equation) 
the left hand side has the kernel as a factor outside the sum; namely, 

LHS = K{^-t) e{a)a{M{^)F{x,,...,Xm;t)). 



On the right hand side of Equation (jlj), before taking the orbit sum, the effect of multi- 
plying by M(x) is 



M(x) - t(^xlxl . . . x'^^FiO, X2, X3, . . . , x^; t) + 





(5) 



+ XiX2xl . . . x'^Fi^Xi + X2, 0, X3, . . . , x^; t) 

+ . . . 

-|- XiX2X^ . . . X^_-^Xjy^ F(^Xi, • • • J 2^m— 2) ~l~ ■^mi 0, t) 



Note that the coefficient of F{0, X2, x^, . . . , Xm', t) is split because 



s 



s — Xi + Xi 



1 + 



s — 



s — 



s — 



which is easier to manipulate when the orbit sum is taken. Because each of the last m — 1 
terms of the RHS of Equation ([5]) is invariant under aj = + 1) for some j G [m — 1], 
(that is, the generators for &m), by forming the signed sum over &m we reduce these m — 1 
terms to zero, leaving only the first three terms: 



3.4. The constant term extraction operator CT. Our goal is to obtain the series 
F{0, 0, . . . , 0; t). Remark, any term in F{xi, . . . , Xm', t) containing non-zero exponents of Xj's 
for i G [m] disappears when Xi is set to 0. The exponents of each {xi + Xj+i + ■ ■ ■ + + 1) 
are all non-negative, implying that to get a constant term, each factor in parentheses must 
go to the constant, leaving only the variable t, keeping track of the size of the partition. For 
the sake of brevity in presentation, we define a linear operator for constant term extraction, 
namely [x^a;^ . . . x'^]F. 

Definition 2. Let CT be the constant term extraction operator defined on Laurent series 
by the following action on monomials: 



(6) K(x;t) J2 e{a)a{M{^)F{x^,...,x^-t)) 



•^2'^3 • • • •^m-^(O) "^27 • • • ) 2^7711 ^)) 





CTixl'xl^ . . . 




0, if Cj 7^ for some i G [m] 
otherwise. 



9 



Before applying our constant term extraction operator CT, to the orbit sum, Equation (|6]), 
we first divide Equation by M(x)_ft'(x; t): 



(7) E 



e(a)a(M(x)F(xi,...,x^;t)) 



M(x) 



^ e(a)a(M(x)) 



1 - t/is V ^ M (x 



^ ^ e(o>r(x|x|. . .x™F(0,a;2, . . . 



ctG6„ 



\ / \ a:2H \-Xm+l ' 



M(x) 



On the LHS of Equation ([7]), after CT is apphed, only the term corresponding to o = id 
remains, yielding 

Cr(F(xi,...,x^;t)) = J] tW 

7rgn('") 

because the other terms all contain a nonzero exponent for some Xi where i G [m] . When we 
extract the coefficient of t" from CT(F(x;t)), we get precisely the number of set partitions 
of size n without an (m + l)-nesting. 

The task is now clear: We need to extract the coefficient of x\x\ . . . from the RHS 
of Equation ([7]). Fortunately, since CT is a linear operator, we can examine the RHS of 
Equation ([7]) term by term, namely, by considering the three surviving orbit sums one at a 
time. We illustrate this process with an example using nonnesting set partitions. 



4. 2-NONNESTING SET PARTITIONS 

The generating function derivation in this section is for pedagogical purposes, to illustrate 
how to manipulate Equation ([7]). Indeed, there are easier ways to determine the generating 
function for the Catalan numbers. 

To enumerate 2-nonnesting set partitions we set m = 1 in the above equations. In this case, 
F{Q;t) = ^^gn(i) which we rewrite as F{0;t) = '^Fnt"'. Since m = 1, the associated 
symmetric group is ©i which only contains the identity permutation; thus the functional 
equation is: 

(8) F(x,<) = ^ (i + l)F(0;*). 

1 - i + ij (xi + 1) 1 - < + l) (li + 1) Va:i J 

Though this is an easy case, writing out the action of CT shows us how terms are collected 
and coefficients computed in a rather slow way. First expand Equation ([8]) as power series 
to get 

(9) F{xr, t) = + 1)"+^ ( - + 1 I t-- E((^i + l)^)"""' ( - + X E 
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Now apply CT to get 

E ^ cm., t)^Y.Y. (" : (" - E E (" : X E 

n=0 n j=0 V-^ ^ / V"'/ „ j=0 ^ n=0 

(" / 1 i\ / \ n*+l / * , i \ 2 

j=0 ^-^ / \J/ „,=o j=0 V / 

We deduce a recurrence for after simplifying the binomial summations: 
When all F^'s are collected to the left, we get 



2\n + l I ^ \j + 



to- 



_ 1 /2n + 2 
''~2\n+l 



k=0 

Upon noticing that the left hand side is a convolution product, we define 

°° '2k\ , 1 



/'2k\ 

^ ^ 



X 



to obtain 



1 1 

f{x)g{x) = —{g{x) -1), or /(x) = — (1 - Vl^4x), 



the famous Catalan series as expected. 



5. 3-NONNESTING SET PARTITIONS 

The first non-trivial case is 3-nonnesting set partitions to study how the orbit sum produces 
sums of products of multinomial coefficients. Notice that the previous example with m = 2, 
no explicit formula for was used; instead, it the F^s was defined in terms of all previous 
Fj for all j < n. However, the convolution product allowed a successful isolation of the 
generating function f{x). For this reason, it is our opinion that the study of the structure of 
convolution- like product for m = 3 and beyond may shed light in the nature of the generating 
series. As in the previous example, through the investigating of the action of CT on the RHS 
of Equation ([7]), we show how the conditions of summation indices turn out to reduce to a 
simple equation, thus restricting the degree of freedom. This exercise, though tedious when 
carried to the next case, m = 3, lends evidence to the conjecture by Bousquet-Melou and 
Xin in [1] that the generating function of the 4-nonnesting case is not D- finite. Furthermore, 
we get enumerative formulas as functions of the label, and an understanding of the structure 
of the generating functions. 
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5.1. First term of RHS of Equation ([7]). The first extraction is resolved by a simple 
combinatorial argument on the total way to combine the exponents to get a constant: 



/Xi + X2 + lv^ a{xixl) 
1-ths ^ '^""^-^ 

cre&2 ^ 



c^((i:(^-^i;H-i)"(......ir«.")(i-|)) 

^ n=0 



n=0 0<h,i2,«3 



n=0 0<Zi,i2,i3 



^1) ^2; hj \ ^2 + 1 



h+l2+h=n 



5.2. Second term of RHS of Equation ((Tj). The remaining terms are expressed in terms 
of 



00 n+l 

F(0, t)= Yl (^2 + = E E + ^2)'^", 

7rgn(2) n=0 k=l 



where Fn{k) is the number of partitions of [n] in 11'^^) such that 02 (tt) = k. Remark, we have 
the relation = Xlfcii Fn{k)k by the comment that each partition n has 02 (vr) children. 
Under the action of ©2, the orbit sum has two terms, one from the identity and one from 
interchanging Xi and X2'- 



(xi + X2 + l)t^ a{xlF{0,X2;t)) 
1-ths ^ 

ae62 2 



-+^+0 ((^i+^2+i)tr^) X (-F(o, 0:2; - 5^(0,0:1; t)). 

\ _ \Xi X2 / ' X2 ' 



n=0 
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We use the linearity of the operator, and consider this expression in two steps. First, We 
expand this and apply CT to the first term, using the definition of F, to get 



11) cr(V(- + - + l) ((xi + X2 + l)t)"+'-F(0,X2;t) 

n=0 



n=0 0<h,«2,/3, 0<j2J3 

h+l2+h=n 32+h=n-h 



oo n+1 fc / T \ 

n i.— 1 i— n V / 



E*" EE E 

n=0 n*<n-l 0<«i,/2,'3 0<i2,i3 

/i+/2+^3 ="-"■*-! j2+J3=n.-n.*-l-'i 



X 

n=0 fc=l i=0 

72 — n* — l\ / n — n 



X 

fc<r),*+l 



Remark, to extract the constant coefficient with respect to X2-, we impose j2 — h = —i on 
the inner most summation. 

A similar expression is obtained from the second term: 

°° / 1 1 \ " 



n=0 0<h,l2,h, 0<jl,h<n, 
Il+l2+l3=n j\+j-i=n-l-l2 



oo n+1 k / T \ 

EE^'"WEu>'.'" 



E^'EE E 

n=0 n*<n-l 0<ii,/2,/3 0<jrij3 

/l+/2+'3="~n*-l ji+j3=n-n* -2-l2 



X 

n=0 fe=l 1=0 

n — n* — l\/ n — n 



E/..-«(,_^, 

fe<n*+l ^ ' 



where similar conditions as above also apply to surviving terms, namely: — i = ji — /i + 1. 



5.3. Third term of RHS of Equation ([7]). Finally, the action of ©2 on the third term 
of RHS of Equation ([7]) yields two terms as in the previous case, and the analysis is almost 
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identical. The third term is: 

{xi + X2 + l)t / Xixl - 



ths 



V4;^V^l y / V ^2 + 1 X2 Xi + l J 



We take a closer look at the second part: 

n+l 



F{0,X2;t) xiF(0,xi;t) 



X2 + I X2 Xi + 

Applying CT to the entire expression yields 

^n — n* — 1 



n fc=l ^ ' 



(13) E^" E E E 



n*<n-lk<n*+l 0<h,l2,l3 

h+l2+l3=n—n* — l 



3 



/ \ 

^ /n-n*\/'k-l\ ^ / n-n* \/' k-1 

oh,n VI, i2, isj \k - 32) ~ k + 1, J3 J Vi - ii - 1 

\i2+i3='i-n*-h ji+j3=n-»i*-^2-l / 

5.4. A complete expression for F„. We can put the three components together into one 
expression for the coefficient of F„ in terms of Fn*{k), a function of the label where n* < n: 



"^1 / * 1 

Ev-^ ^ n — n —I 

E E 



n*=OA;<n*+l (><h,l2,h 

h+l2+h=n—n* —I 



h-ik, I 



n — n* \ / k \ / n — n* \ f k 



n<, , v^i + 1> i2, js/ v^2 - i2/ Vii> ^2 + 2, is/ v^i - ii - 1 

j2+j3="-n*-l-'i ji+j3=n-i2-2 

f n — n* \ f k — l\ / n — n* \ / k — 1 

^ o|;',3 Vi, J2, Jay V2 - J2) ~ ,1;-,,^ U, k + 1,33) U - Ji - 1 

j2+j3=n-n* -h ji+j3=n-n* -h-l 

Note how F„ is expressed as a convolution-like sum involving all previous F„* for n* < n. 
In this form, the authors are unable to obtain a recurrence for the F„'s. 

6. Complexity of m > 3 

These examples give us a strong flavour of the general formula. The flrst term is always the 
constant term of a rational function, and hence is always D-finite. There arc some sources 
for added complexity when m is greater than 2. First, the number of terms in the orbit 
sum grows like m!, although one can expect them to be of a similar form, as was the case 
in the m — 2 case. The number of parameters that play a role in the formulas is perhaps 
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the key difference. In the m = 1 case, we ehminate dependence on the parameter, and 
determine direct recurrences. In the m = 2 case, we use the parameter 02, but we also have 
the additional property that the sum of this parameter over all 3-nonnesting partitions of 
size n is the number of 3-nonnesting partitions of size n + 1. Thus, there is an additional 
relation. 

We avoid the full treatment of the m = 3 case, and rather go directly to the typical 
effect of CT on the orbit sum to illustrate how this expression is increasingly complex. The 
functional equation in this case is 

Xi + .1-2 + .r.-i + 1 \ ^ , , aixix^xl) 

e cr 



^1^) — r—fhi — 



1 — ths ^ X\xi,x\ 



1 — ths ^ xxxixi 
(TG63 2 ^ 



+ a;2 + + 1 v-^ e(cr) ( x\x\F{^,X2-,X2,\t) 

rCr ' 



1 — ths ^ X\X%x\ V + 2^3 + 1 

cTe©3 ^ ^ J \ 



Applying CT to the first term yields a sum of six multinomial summations that simplifies to 
an expression of the form 



E E 

fc=0 0<Zl,Z2,«3,«4<fe, 



k 

h, h, h, I4 



R{h, hi h, U), 



where i? is a simple rational function, and, as we noted earlier this expression is D-finite, 
since it is a coefficient extraction of a rational function. 
The second and third terms involve 

F{0,X2,X3;t)^ J2 (xs + xa + ir^'^n^^s + ir^"^""'^"^^'"'. 
7ren(3) 

The result is that when C is applied to the second and third terms, we get the nested 
summations involving complex expressions of 02 (tt) and 03(77). The following is a typical 
sample expression for the coefficient of 



E E E 

k<n-l0<h,l2,l3,l4<k, ^ ' ' ^ 0<j2,j3,j4<k 

Il+l2+l3+h=k Il+j2+j3+j4=k 



k + 1 

h + I,j2,j3,i4 



Vitt 



E E 

\=n-k-l 



02(7r) \ ^ /as (tt) -02(77) 

k - J2, P, a2(7r) -p -12+32) \ h-p- J3 
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n 



m 


OEIS # 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


1 


A000108 


1 


2 


5 


14 


42 


132 


429 


1430 


4862 


16796 


58786 


208012 


742900 


2674440 


9694845 


2 


A108304 


1 


2 


5 


15 


52 


202 


859 


3930 


19095 


97566 


520257 


2877834 


16434105 


96505490 


580864901 


3 


A108305 


1 


2 


5 


15 


52 


203 


877 


4139 


21119 


115495 


671969 


4132936 


26723063 


180775027 


1274056792 


4 


A192126 


1 


2 


5 


15 


52 


203 


877 


4140 


21147 


115974 


678530 


4212654 


27627153 


190624976 


1378972826 


5 


A192127 


1 


2 


5 


15 


52 


203 


877 


4140 


21147 


115975 


678570 


4213596 


27644383 


190897649 


1382919174 


6 


A192128 


1 


2 


5 


15 


52 


203 


877 


4140 


21147 


115975 


678570 


4213597 


27644437 


190899321 


1382958475 



Table 1. Numbers of set partitions of n avoiding an m + l-nesting. The OEIS 
numbers refer to entries in the Onhne Encyclopedia of Integer Sequences [1] 



7. Computing series expansions 

All three Equations ([T]), ([3]), and (jlj) are used to generate initial terms in the series. To 
improve convergence, we slightly modify the x equation: 

, X2 ) • • • ; Xfn ) t) = S + SthF{xi, X2,. . ., Xm] t) 

S F{0,X2,X3,. . .Xm);t) 



(17) \S-Xi Xi 



m 



J=2 



X 



J 



Notice that in Equation fll7p . if one has a series expansion of -F(x; t) correct up to t^, then 
substituting this series into RHS of Equation ( JT71) yields the series expansion of F correct 
to t^"*"^ because the RHS of Equation f|T71) contains a term free of t, otherwise, the degree of 
t is increased by 1. We have thus iterated Equation ([T7|) to get enumerative data for up to 
m = 9. 

For 3-nonnesting set partitions, an average laptop running MaplelS can produce 70 terms 
in a reasonable time (less than 24 hours). For m = 4, only 38 terms; m = 5, 27 terms; 
m = 6, 20 terms; m = 7, 16 terms, m = 8, 12 terms; and finally m = 9, 12 terms. The 
limitation seems memory space due to the growing complication in the functional equation 
when m gets larger. 



8. Conclusion 

Without passing through vacillating lattice walks or tableaux, the generating tree ap- 
proach permits a direct translation to a functional equation involving an arbitrary number 
of catalytic variables satisfied by set partitions avoiding m + 1-nestings for any m. Constant 
term coefficient extraction analysis gives us insight into why the number of 3-nonnesting set 
partitions should be more easily controlled than those of higher non-nesting set partitions. 
The authors are aware of the techniques developed for constant term extraction and are 
investigating how such techniques can give insight to the analysis of m-nonnesting numbers 
of set partitions. Though explicit generating trees are given, formulas thus generated still 
depend on labels of set partitions. Perhaps further study into the nature of generating trees 
which give rise to D-finite series, along the lines of the study in |[2] will help us understand 
the differences. 
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A second way that might yield a proof of non-D-finiteness would be to use our expressions 
to determine bounds on the order and the coefficient degrees of the minimal differential equa- 
tion satisfied by the generating function. Though a tantalizingly simple idea, the limitation 
seems still the lack of data when larger m's give so few values relative to the number one 
would need to test non-D-finiteness. Nevertheless, this would guide searches and a fruitless 
search would then be a definitive result. 

Finally, our generating tree approach is limited only to the non-enhanced case. For a more 
general treatment of the subject involving enhanced set partitions and permutations, both 
enhanced and non-enhanced, we refer the reader to by Burrill, Elizalde, Mishna, and Yen. 
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